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The 1968 American Educational Research Association (AERA) Presession, 
Instructional Product Research, was held at the Chicago Sheraton 
Hotel from February 3-7, 1968. The presession provided an opportunity 
for the tryout and evaluation of product-development training materials 
produced under the Staff Training project at the Southwest Regional 
Laboratory for Educational Research and Development. 

The 1968 AERA Presessions were supported by a grant from the United 
States Office of Education. 
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INTRODUCTION AND OBJECTIVES 



The past two years have brought a sharp upsurge of educator interest in 
the procedures associated with the systematic development of educational pro- 
ducts that achieve pre-specified instructional objectives. A concomitant of 
the recent support for the development and evaluation of new instructional pro- 
grams has been an increased demand for skilled product developers. Yet, due in 
part to the recency of the product development movement, individuals with special 
competence in this area are in short supply. 

The overall goal of the 1968 AERA Presession, Instructional Product Research , 
was to develop the participants' skills in product development and research. The 
procedures used in planning, conducting and evaluating the presession parallel 
those employed in programmatic product development efforts. Initially, the 
specific competencies required for the conduct of successful product development 
and research operations were carefully analyzed by the presession staff. The 
instructional objectives and evaluation procedures for the presession were de- 
rived from this analysis. Instructional materials and activities designed to 
implement the pre-specified objectives were then either prepared by the presession 
staff or selected from among product-developer training materials produced at 
the Southwest Regional Laboratory for Educational Research and Development. 

The analysis of important product development skills yielded a list of 12 
instructional objectives. Participants who mastered these objectives were 
able to perform the following tasks at the conclusion of the five-day presession: 
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1. Identify and specify the objectives for instructional products in terms 
of observable learner behavior. 

2. Classify instructional objectives according to a modified, four-category 
version of the Taxonomies of Educational Objectives . 

3. Discriminate among instructional objectives which contain (a) no minimal 
levels of learner behavior, (b) minimal levels of behavior only for an 
individual learner, and (c) minimal levels of behavior for a group of 
learners . 

4. Select and/or write suitable test items, given precise instructional 
objectives . 

5. Correctly classify (according to a four-category scheme) different types 
of criterion measures which may be used to evaluate educational products. 

6. Correctly classify different components of sample instructional specifi- 
cations. 

7. Sequence enroute behaviors according to prescribed criteria. 

8. Correctly identify instances in which product development rules have 
been (a) followed or (b) violated when presented with a series of 
fictitious vignettes describing product development activities. 

9. Identify in given instructional products examples of selected develop- 
ment principles. 

10. List and briefly describe appropriate procedures in product tryout and 
revision. 

11. Specify criteria useful in evaluating the effectiveness of product 
development operations. 

12. Recommend appropriate research designs for hypothetical situations 
requiring the experimental evaluation of instructional products. 



INSTRUCTIONAL PROGRAM AND MATERIALS 



The instructional program and materials were organized to provide direct 
instruction and practice on the presession objectives. Instructional sessions 
were scheduled daily from 9:00 a.m. until 4:30 p.m., and evening meetings with 
optional attendance were held from 7:00 until 9:00 p.m. on the first three nights 
of the presession. 

The complete schedule for the presession is presented in Table 1 . 




PRESESSION SCHEDULE 
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The da'^time program was divided into four sessions daily. Typically, direct 
instruction and practice were provided on the presession objectives during the 
first three sessions each day. During the final daytime session each participant 
worked individually on instructional handouts given that day, prepared his own 
small-scale instructional product, or consulted with presession staff members on 
his ongoing or planned product development and research activities. 

The optional evening meetings were scheduled to provide participants with 
an opportunity to interact with each other and the presession staff in group 
discussions. Tape- films trip instructional programs were shown at the beginning 
of the first two evening meetings to stimulate the discussions. Attendance at 
the optional evening meetings ranged from a low of 25 individuals (53% of the 
47 presession participants) at the initial session to a high of 40 participants 
(85%) at the final meeting. 

A total of 38 separate handouts, ranging in length from one to 60 pages, 
comprised the materials used for instructional and evaluation purposes. A 
complete listing of the materials provided for all presession participants is 
presented in Table 2. 



EVALUATION RESULTS 

The primary evaluation source was the post- instructional performance of the 
participants on the instructional objectives of the presession. Two parallel 
test forms containing 190 items each were employed as direct measures of the 
12 objectives. Each test form measured all 12 objectives, and each form con- 
tained the same number and type of items for any single objective from among 
the twelve. The pretest, Form X and Form X -Part II, was administered at the 
opening session on the first day of the presession. T^e post-test, Form Z and 
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Table 2 

PRE SESSION MATERIALS 



1. Pre Tests: Form X, Parts I and II (20 pages & 5 pages) 

2. Pre Test Correct Answer Key (1 page) 

3. Profile Sheets (1 page) 

4. Instructional Product Research: Introductory Hand-outs (8 pages; 

5. Educational Objectives Program Answer Sheets (1 page) 

6. Objective 3 Practice Sheets (1 page) 

7. Product Documentation and Review Guidelines (10 pages) 

8. Instructional Objectives (17 pages) 

9. SWRL Technical Glossary (8 pages) 

10. Establishing Performance Standards (17 pages) 

11. Selecting Appropriate Educational Objectives (19 pages) 

12. Product Research: A New Curriculum Specialty (5 pages) 

13. Improved Educational Program Answer Sheets (1 page) 

14. Improved Educational Program Criterion Tests (1 page) 

15. Criterion Measures Examples (1 page) 

16. Educational Criterion Measures Answer Sheets (1 page) 

17. Objective 6 Practice Sheets (1 page) 

18. Rules for the Development of Instructional Products (60 pages) 

19. Design Specifications: Objectives and Prototype Items (21 pages) 

20. Writing Instructional Specifications (15 pages) 

21. Educational Criterion Measures (24 pages) 

22. Write is Right (5 pages) 

23. Sequencing Enroute Behaviors (25 pages) 

24. Exams; Research in the Schools - Part I (4 pages) ^ 

25. Answer Sheets: Classifying Educational Research Studies (1 page) 

26. Criterion Test Items: Objective 12 (1 page) 

27. Appropriate Practice (17 pages) 

28. Providing Knowledge of Results (17 pages) 

29. Make it Interesting (14 pages) 

30. Avoid Irrelevancies (13 pages) 

31. Exams: Research in the Schools - Part II (6 pages) 

32. Answer Sheets: Interpreting Research Results (1 page) 

33. Simplified Designs for School Research (26 pages) 

34. Developing the "D" in Educational Research and Development (11 pages) 
35*. Research, Development, and Improvement in Education (16 pages) 

36. Post Tests; Form Z: Parts I and II (22 pages & 5 pages) 

37. Post Test Correct Answer Keys (1 page) 

38. AERA Evaluation Forms (1 page) 
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and Form Z - Part II, was administered during the first period on the final 
day. A copy of Form X is included as Appendix B of this report. Form Z is 

available from the authors upon request. 

Table 3 shows the pretest and post-test mean scores by objective and 
presents a group profile for each test. The table reveals that the mean pretest 
score for all participants on the 190- item test was 112.6 (59%) and the mean 
post-test score was 155.7 (827o) . Performance gains by objective from pretest 
to post-test ranged from 8% (objective 2) to 73%> (objective 12). As indicated 
in the table, the participants attained a group post-instructional performance 
level of above 80% on 9 of the 12 objectives. 

The presession critique forms prepared for use at all 1968 AERA presessions 
and completed by the participants and staff at the conclusion of the presession 
served as additional sources of evaluation. A list of the 47 participants and 
certain descriptive data about them are presented in Appendix A, and the responses 
of all participants who completed the Participant Evaluation Form are summarized 
in Appendix C. Staff responses are shown in Appendix D on the form entitled 
Presession Critique for Staff Members. 

The summarized responses of both participants and staff members show con- 
sistently favorable evaluation of all aspects of the presession. Tabulation of 
positive and negative responses and comments on the Participant Evaluation Form, 
with items related to the hotel facility omitted, reveal a ratio of 10 positive 
responses from participants to each negative response. 

Based upon the post-instructional achievement of the participants and the 
written evaluations from both participants and staff members, it appears 
reasonable to conclude that the presession was a success. 




Table 3 



*GROUP PROFILE: MEAN PRETEST AND POST-TEST SCORES BY OBJECTIVE 

Pretest (Form X) 
Post-test (Form Z) 

Obj. Exam No. No. 

No. Content Section Items Right Percent Correct 











Pre 


1 


Instructional Objectives 


II 


10 


7.2 


2 


Objective Domains 


III 


10 


8.9 


3 


Performance Standards 


IV 


10 


5.9 


4 


Prototype Test Items 


VI 


10 


7.0 


5 


Criterion Measures 


V 


10 


6.1 


6 


Instructional Specs 


VII 


10 


5.3 


7 


Sequence Behaviors 


XV 


10 


1.0 


8 


Development Process 


I 


50 


38. 6 


9 


Development Principles 


VIII- 

XI 


40 


30.5 


10 


Tryout and Revision 


XIII 


7 


1.3 


11 


Development Operations 


XIV 


18 


.J, 


12 


Research Designs 


XII 


5 


. 6 




TOTALS 


all 


190 


112.6 



Post 0 10 20 30 40 50 60 70 80 90 100 




^vEhe table shows the mean number of correct answers by objective for all 
participants who completed each test. The broken (left-hand) profile 
line in the 'percent correct' column indicates pretest performance: 
the solid (right-hand) line shows post-test performance. 
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APPENDIX A 



PARTICIPANTS 




PARTICIPANTS 



An approximate total of 75 completed applications were received for the 
Instructional Product Research Presession. Letters of acceptance were mailed 
to the first 56 applicants. All subsequent applications were rejected because 
the only available meeting room at the presession hotel could reasonably accom- 
modate a maximum of 50 people. Withdrawal by some accepted applicants prior to 
the presession and the failure of others to appear at the meetings reduced the 
number of actual participants to 47 . A list showing the name, address, insti- 
tutional affiliation and job description for each participant is presented in 
the remaining pages in this appendix. 

To supplement the list of participants, additional descriptive data were 
tabulated from the applications. These data are presented in the 8 numbered 
topics immediately below. 

1. Average age of the 47 participants: 41 

2. Sex: 35 males, 12 females 

3. Number holding doctorate: 36 

4. Number who have had one or more funded projects: 22 

5. Average number of funded projects for all participants: 1.1 

6. Number who have had one or more publications (as determined by item 14 

on application form): 37 

7. Average number of publications for all participants: 8.3 

8. Institutional affiliation 

a. Colleges and Universities: 31 

b. Regional Laboratories: 6 

c. Public Schools: 6 

d. Business and Industr}^: 3 

e. National Educational Research Bureau 

(Director of Educational Research, Sweden): 1 
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LIST OF PARTICIPANTS 

AERA PRESESSION NO. 8 
Instructional Product Research 
February 3-7, 1968 



Name 


Address 


Affiliation 


Title or 
Position 
Description 


Allen, William H. 


Dept, of Instructional Technolgy 
School of Education 
University of South. California 
Los Angeles, 90007 


University of 

Southern 

California 


Professor of 
Education 


Berman, Marlene 


3750 Woodward Avenue 
Detroit, Michigan 48235 


Michigan-Ohio 
Regional Edu- 
cational Lab. 


Research & 
Teaching 


Bernazza, Ann Marie 


U-93 University of Connecticut 
Department of Ed. Psychology 
Storrs, Connecticut 


University of 
Connecticut 


Research 

Associate 


Bing\«an, Richard M. 


10601 Ease 65th Street 
Raytown, Missouri 64133 


Mid- continent 
Regional 
Educational 
Laboratory 


Educational 

Program 

Specialist 


Blaney, Jack P. 


Extension Department 
University of British Columbia 
Vancouver 8, B.C. Canada 


University of 

British 

Columbia 


Assoc. Dir. of 

University 

Extension 


Broadbent, Frank W. 


College of Education 
Drake University 
DesMoines, Iowa 


Drake University 


Assoc. Professor 
of Education 


Carter, Heather L. 


7401 New Hampshire Avenue, #907 
Hyattsville, Maryland 20783 


University of 
Maryland 


Research and 
Teach iiig 


Carr, Julian W. 


800 Washington Avenue 
Minneapolis, Minnesota 


Readers Digest 

Educational 
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APPENDIX B 



CRITERION TESTS 

EXAMINATION: THE DEVELOPMENT OF INSTRUCTIONAL PRODUCTS 



FORM X 



Form X and Form X - Part II, presented in this appendix, served 
as the pretest. The posttest. Form Z and Form Z - Part II, are 
available from the authors upon request. 

The first section of both the pretest and posttest were originally 
prepared for staff-training purposes at the Southwest Regional 
Laboratory for Educational Research and Development. 



Examination: The Development of Instructional Products 

Form X 

* 

W. James Popham 



General Directions: Complete only the sub-tests below which are checked , 
as directed in the examination booklet. You may write on the examination 
booklet itself, but please make all of your responses on the answer sheet 
which has been provided. Be sure to write your name on the answer sheet. 



I. 

IX. 

III. 

IV. 

V. 

VI. 

VII. 

VIII. 

IX. 

X. 

XI. 



THE DEVELOPMENT PROCESS 
INSTRUCTIONAL OBJECTIVES 
OBJECTIVE DOMAINS 
PERFORMANCE STANDARDS 
CRITERION MEASURES 
PROTOTYPE TEST ITEMS 
INSTRUCTIONAL SPECIFICATIONS 
APPROPRIATE PRACTICE 
KNOWLEDGE OF RESULTS 
PROMOTING LEARNER INTEREST 
AVOIDING IRRELEVANCIES 



PART I. THE DEVELOPMENT PROCESS 



Part I. Directions : This part of the examination consists of five brief 

descriptions of segments in the product development process. Each descrip- 
tion is followed by ten statements, some of which are correct. You are to 
read each description, then on the answer sheet which has been provided, 
mark an A for each statement which is correct . Nothing needs to be marked 
for a statement which is incorrect. If insufficient information has been 
presented for you to judge the correctness of a statement, leave the item blank. 

Generally speaking, certain incidents in the development of instructional 
products are recounted in the fictitious descriptions. Your task is to 
identify correct procedures which were employed or errors which were 
made by the product developers. No attempt has been made to be devious 
in the examination. You need not ’’read between the lines*’ in order to 
judge the correctness of the 50 statements. The answers should be apparent 
to you if you are familiar with appropriate steps in the product development 
process. Be sure to use the answer sheet for your responses. Now commence 
with the first description and its accompanying ten statements. 



* 

Parts IV and VI were 
by Robert J. Berger 



prepared by Eva L. 



Baker . 



Part VII was prepared 



F 
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Exercise One 

It is mid-December and Frieda, a regional laboratory employee, has been 
given the responsibility of developing a short self-instruction program to teach 
sixth grade pupils how to use commas. She his received a set of five explicit , 
instructional objectives from the individuals who originally formulated the 
project. Along with these objectives there are samples of a single prototype 
test items for each objective. Frieda has been told that the instructional 
product is to take no more than four hours of the average learner s time. 

The first thing she does is to develop a 40-item criterion test to be used 
vit the close of the program. She has each of the five objectives represented 
by at least five test items, although two objectives which she feels to e more 
important are represented by 10 items. 

Frieda carefully considers the enroute (intermediate) behaviors which the 
learner must master on his way to the criterion behaviors and then sequences 
these from least to most difficult, ending with behaviors equivalent to those 
called for in the instructional objectives. She then prepares practice sequences 
for each of the enroute and terminal behaviors so that the learner will be 
to practice using commas in a variety of situations. After having three colleagues 
react to her first version of the instructional product, she makes a number of 

revisions . 

Frieda then arranges to field test the program in the public schools and 
securL the cooperation of a nearby elementary school. She arranges to use 
three classes for approximately one week and administers the program in early 
February. At its conclusion the 40-item criterion test is given to each of 
the 86 children who completed the program. 

Frieda is pleased that all youngsters were able to finish the instructional 
product in three hours or less. She is somewhat concerned, however, that the 
average score on her 40-item test is only 21.2 correct. She resolves to study 
the pLt-test results as well as the responses made by pupils during the program 
and to make the revisions which seem dictated by the data. 

^********* 

Which, if any, of the following statements are correct? (Mark on the 
answer sheet an A for any which are correct.) 

1. Frieda was probably correct in providing Practice behaviors for 
the learner which were equivelent to those calle for 
instructional objectives, 

2. She should have required at least two prototype test items along 
with the instructional objectives, 

3, Each objective should have been represented equally on the criterion 
test. 

4. Frieda field-tested her first version of the product on too many 
learners, 

5, She should have pre-tested the subjects, 

6, Her program was dull. 
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7. Too much time was taken to complete the first version of the product. 

8. Five objectives are too many for such a program. 

9. The enroute behaviors should not have been sequenced from least 
to most difficult. 

10. More than three colleagues should have reacted to the first version 
of the program. 



Exer^cise Two 



A group of three beginning product developers has been assigned the general 
task of preparing a 50-minute self-instruction sequence for learners in the 
fieu: of English. The resulting product will be used by junior high school 
. '•'■r'.cccrs, more specifically, ninth grade English teachers, as a remedial 
program fcr students who are having difficulty with one or more topics in the 

■VC-jLOt. 



After some discussion (approximately three days) among themselves and a 
few experienced instructional programmers, the three decide to develop a 
ron--dial program which will increase the student's ability to diagram standard 
•sentences so that each of the eight parts of speech are clearly identified. 

They then turn to the task of deciding on an appropriate criterion test, 
spending the next month in a series of carefully organized discussions re- 
garding the possible methods of assessing a learner's ability to manifest 
mastery of the diagramming process. For example, during one meeting the three 
i-civ-iduals develop sample items and take turns answering them, then analyzing 
Che adequacy of each other's responses. The following objective is agreed upon 
iir the r‘"'gram; 

When presented with 10 previously unseen sentences, five simple (that is, 
one clause) and five complex (that is, two or more clauses), the learner will 
h<’ able to diagram at least eight without any errors according to the procedure 
■p.icified in the 50-minute instructional program. Ninety per cent (or more) 

*'Te learners who complete the program must satisfactorily achieve this level 
of proficiency. 

The terminal behavior having been selected, the three product developers 
u prepare 25 test items based on the above objective and arrange to administer 
the 25-item test to a group of 30 ninth grade English students. Gratifyingly , 
the itudents are not able to perform well on the test, so the three developers 
next address themselves to a careful task analysis in order to identify ne- 
cersary types of entry behaviors (the skills the pupil possesses before starting 
the program) and enroute behaviors (the intermediate skills the learner must^ 
achieve in order to attain the terminal behavior). Having done this, approxi- 
mately four days being expended on the task, an appropriate pre-test is pre- 
pared including items which reflect desired (1) entry behaviors, (2) enroute 
behaviors, and (3) criterion behaviors. This test is then administered to ten 
pupils and, having carefully analyzed their responses, the product developers 
begin to prepare the first draft of their instructional product . • . 



Which, if any, of the following statements are correct? 

11. The preparation of the criterion test quite properly preceded 
the development of the instructional product, 

12. It was unnecessary to assess learners' ability to perform entry 
and enroute behaviors. 

13. Too much time was spent in the determination of a suitable 
instructional objective. 

14. The instructional objective decided on was not sufficiently precise. 

15. The minimum level of proficiency expected of learners was not 
well specified. 

16. The adequacy of the original formulation (i.e., the selection of 
the particular topic) was not well justified. 

17. Too much time was spent by the product developers on the original 
selection of a topic. 

18. The expanded pre-test, the one with the addition of items based 
on entry and enroute behaviors, should have been given to at 
least 30 learners. 

19. The first version of the instructional sequence should have been 
prepared and tried out prior to the administration of the pre-test, 

20. It was quite appropriate for the product developers to undertake 
the task analysis of entry and enroute behaviors. 



Exercise Three 



Fred Peabody, an experienced instructional programmer, has been assigned 
the responsibility of developing a self-instruction program to teach third grade 
youngsters how to add and subtract fractions. One of the first things he does is 
to go to a third grade classroom and ask permission to talk with six of the 
youngsters. Individually he discusses the topic of fractions with each of them 
and attempts to find out what the students already know that will be relevant to 
his task. Having completed his assessment of the data secured from these inter- 
views, he devises the following objective for his program: "At the conclusion 
of the program the learner will manifest a sophisticated ability to handle addi- 
tion and subtraction problems involving fractions." 

After consultation with the teachers and the administrators he prepares a 
formulation paper in which he identifies the above objective and attempts to 
support the value of the proposed program. He cites the opinions of teachers 
who indicate that a short-term program such as that which he proposes will have 
considerable utility in their classes and cites evidence from published bibli- 
ographies of program material that there is currently no such short-term program 
available to teachers in a form which does not require the use of teaching 
machines. Accordingly, he proposes that his program will take approximately 
one and a half or two hours to complete and will be presented by printed-paper 
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booklets. This formulation paper is presented to a number of colleagues with 
whom Peabody works and, having read it, they agree that his selections of topic 
and presentation medium are sound. 

He develops a criterion test in which the learners must add and subtract 
pairs of fractions. He also prepares other items which assess the learners 
ability to add and subtract whole numbers. He combines these to form a pre- 
test which he administers to a sample of 25 third graders drawn from a nearby 

elementary school. 

Having determined that the youngsters cannot perform the terminal behavior 
but do possess the desired prerequisite skills, Mr. Peabody then prepares an 
early version of his program. 



Since he employs a variety of approaches in his programming efforts, Mr. 
Peabody attempts to select a tactic for this particular product. After intro- 
ductory remarks and a certain amount of exposition (approximately three 
worth) he gives the student a series of ten sets of simple fractions to add. 

After they have concluded adding this set, he provides them with the correct 
answers which they can see by turning a page and comparing the proper answers 
with the answer sheet on which they have been instructed to make their responses. 
Several more sets of these addition exercises are provided in order that the 
learner may practice the operation of fraction addition. After each ten problems 
Mr. Peabody provides knowledge of results in a manner similar to that described 

above. 



For variety's sake, however, when it comes to subtraction of fractions, he 
approaches it in an entirely different fashion. He presents a series of short 
story (word) problems to the learner which involve the subtraction of fractional 
quantities. The learner is obliged to "think through" what fractional quantitie. 
are required and then form a mental subtractional operation regarding each of 
these particular problems. After each set of three word problems, once more 
’-.nowledge of results is provided whereby the learner can check the accuracy of 
his responses. In all, five sets of these subtraction word problems are pre- 

sented. 

Mr. Peabody has multiple copies of his program prepared on a mimeograph 
machine and takes it to the same elementary school where 
such excellent cooperation. He administers the program at 
he had his original interviews, using again the same students that were so 
helpful two weeks earlier and two other classes as well. At the conclusion 
of the program which, as he predicted, took approximately an hour and 45 minutes 
for most sLdents to complete, he administers the series of 20 simple additi^ 
problems and 20 simple subtraction problems, each involving two fractions, 
the 40-item test he is disappointed when the mean performance of the 28 
who complete the program pre-test is only 26.3. Mr. Peabo y eci ° . 

his program consistent with these data. He is particularly anxious to check 
the responses which learners made during the program since he believes these 
will provide him with clues as to what sections are most in need of alteration. 



* * * ********* 
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Which, if any, of the following statements are correct? 

21. Mr. Peabody wisely planned to use learner responses during the 
instructional program as a guide for improving his program. 

22. The performance of Mr. Peabody's students on his program was 
acceptable and he ought to be satisfied. 

23. Mr. Peabody did not define the terminal behavior adequately in 
his objective. 

24. Too much delay may have been involved in the knowledge of results 
provided for the majority of the learners' responses. 

25. The programmer should not be concerned with "variety" in 
developing his instructional materials. 

26. Mr. Peabody should have involved more learners during his initial 
exploratory interviews. 

27. He probably field tested his first version of the program with 
too many learners. 

28. He should have field tested his program with a group of students 
other than those involved in his initial exploratory interviews. 

29. The operations associated with his formulation procedure were 
inadequate. 

30. Peabody had too few items on his criterion test. 



Exercise Four 



Mrs. Shear has acquired a reputation in the past several years of being 
a remarkably skilled product developer. She believes that much of her reputation 
is due to the fact that she has developed a workable procedure for accomplishing 
instructional objectives. She has discovered that the preparation of audiotape 
narration, coupled with the use of visual transparencies placed on an overhead 
projector by the teacher, efficiently accomplishes the behavior changes she 
desires. Furthermore, she always tests her program through the use of five 
alternative multiple choice examinations which she has become most adept at 
constructing. No matter what the objective, whether cognitive, affective, or 
psychomotor in nature, and no matter how complex, Mrs. Shear analyzes it in 
such a way that it can be handled through the use of this audio tape-trans- 
parency approach. 

The first thing she always does is construct a relevant multiple choice 
test. She uses five choice items because of their greater efficiency in dis- 
criminating between the more and less knowledgeable learners. She is careful 
to try out her tests with an appropriate group of learners so that she can tell 
which items properly discriminate between the more and less knowledgeable 
learners in the group. Although sometimes her methods fall short of expectations, 
Mrs. Shear's reputation as a productive programmer is widely held among her 
colleagues . 




************ 
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Which, if any, of the following statements are correct? 

31. Mrs. Shear quite appropriately prepares her criterion test 
before developing the instructional materials. 

32. Mrs. Shear is too inflexible regarding her selection of programming 
strategies. 

33. Her approach is bound to fail with highly creative youngsters. 

34. Few teachers would be willing to place transparencies on an 
overhead projector as instructed in the program. 

35. There are some criterion behaviors for which multiple choice 
tests are inappropriate, hence, Mrs. Shear ought not to use them 
in all situations. 

36. Mrs. Shear's tests are inappropriate because they are built on a 
norm referenced (comparing student to student) rather than a 
criterion referenced basis (comparing student performance to goals). 

37. Mrs. Shear's programs are probably uninteresting. 

38. She ought to achieve her objectives in every situation she attempts, 
probably on the first or second draft of her program, or she does 
not deserve the reputation she has as a skilled programmer. 

39. Mrs. Shear should realize that group-paced programs have 
little place in the public schools. 

40. Her programming approach will prove effective only in the 
cognitive domain. 



Exercise Five 

Mr. Smith has been assigned the task of developing a one-week 
(approximately five 50-minute periods) group-paced instructional program 
designed to teach high school chemistry students to treat correctly certain 
analytic equations and problems involving unknown chemical elements. Mr. 
Smith does not attend to the formulation process because this has been 
done by others. His responsibility is to develop the actual instructional 
material. The instructional objective which has been given to him by 
members of the formulation team is the following: 

At the conclusion of the instruction at least 80 per cent 
of the learners will be able to solve seven of ten equation 
problems involving an unknown compound. 

Mr. Smith arranges to talk to several high school chemistry instructors 
and a half a dozen high school chemistry students to secure some ideas as 
to the proper tactics to employ in teaching the particular subject. The 
students are asked how much they already know of the topic so that Mr, 

Smith can identify the competencies he can build upon when preparing 
the instructional sequence. 



He develops four programs j each lasting approximately 40 minutes , 
with the expectation that these early versions will be revised and 
augmented. He adopts a "lean" strategy in programming in which he offers 
the minimum amount of instructional materials that he believes is requisite, 
anticipating that if the program fails it will be easier to add to it 
than to subtract superfluous material from an effective program sequence. 
All of the programmed material is transferred to an audiotape so that it 
can be coordinated with visual materials which are presented on 2x2 
slides. He ultimately plans to transfer the visual sections to a 
filmstrip, but believes the slide presentation will offer more flexibility 
for the subsequent addition or deletion of modified visuals. 

He next develops a criterion test consisting of thirty items in 
which the student is presented with verbal descriptions of chemical 
interactions and asked to describe with chemical equations the nature of 
the quantitative equations which have been verbally described. 

He tries the program with four learners who answer, respectively, 

21, 25, 26, and 27 items correctly on the 30-item test. 

*************** 

Which, if any, of the following statements are correct? 

41. Mr. Smith's expectation that the early version of the program 
will be revised is realistic. 

42. His students did not perform as well as they should have on his 
first draft materials. 

43. A "lean" programming strategy has been demonstrated to be 
ineffectual in this type of task. 

44. Mr. Smith should have prepared his criterion test prior to the 
development of his first version instructional product. 

45. Mr. Smith's instructional materials were probably dull. 

46. The use of audiotape and filmstrip is inconsistent with the 
notion of group -paced program. 

47. Mr. Smith's criterion test was not appropriate for the 
instructional objectives he had been given. 

48. Mr, Smith, or any programmer, has the clear responsibility for 
evaluating the adequacy of the formulation operation no matter 
at what point he is introduced to the development process. 

49. Mr. Smith should not have consulted teachers and pupils prior 

to the development of his first draft of instructional materials. 

50. Mr. Smith should have developed first draft materials which 
were exactly as long as those meant for the program. 



26 



PART II. INSTRUCTIONAL OBJECTIITIS 



P;,rt II Directions: In the following list identify any properly stated instruc 

nSriaTobjectives by marking an A on the answer sheet. Nothing need be marked 
for improperly stated objectives. 



51. 

52. 



53. 

54. 

55. 



56. 

57. 

58. 

59. 



60. 



The student will grasp the significance of the Treaty of Versailles. 

The student will have an attitude favorable to English grammar indicated 

by his response to a questionnaire. 

The student will know six verbs. , . j u 

The student will learn the names of the common tools in wood shop. 

The teacher will list three major causes of the Civil War on the 

chalkboard. . r. i j n t 

The student will know the important battles in World War I. 

The student will prefer cooking to sewing. _ 

The student will be able to correctly thread a sewing machine. 

The student will pay attention as the teacher demonstrates the use of the 

The student will be able to develop a sense of the cultural unity of man. 



PART III. OBJECTIVE DOMAINS 



Part III D irections ; Classify each objective below by marking the correct 
Tetter~ac cording to the following scheme: 



A. 

B. 

C. 

D. 



psychomotor 

affective 

cognitive-"higher than lowest level 
cognitive-" lowest level 



The learner : 



61. is 



62. 



63. 

64. 

65. 

66 . 
67. 



68 . 

69c 

70. 



able to choose the best of two solutions to a geometry problem using 

standards given by the teacher. j 4 .u 4 : 

e:;hibits tolerance for others by displaying good manners toward those of 

minority groups. . 

lists the names and contributions of five key curriculum workers. 

properly knits a baby blanket. 

scores well on the Minnesota Teacher Attitu^ Inventoj^. 
uses instructional principles properly in planning daily lessons. ^ 

plays table tennis according to rules well enough to beat three inexperience 

girls 100 % of the time. 

correctly recites Gettysburg Address from memory, 
scores 80% or better on a spelling quiz. 

displays interest in higher mathematics by volitionally attending lectures 
on this topic 
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PART IV. PERFORMANCE STANDARDS 

Part IV Directions ; For the following objectives, mark A if the^ 
objective has only a student minimal level of learner behavior; ^ if 
the objective has a class minimal level of behavior; and £ if the 
objective has no minimal level of learner behavior. 

71. The class will answer correctly 10 out of 12 multiple choice 
questions on the Roman Empire. 

72. The students will compose an essay on the topic of their summer 

vacation. , 

73. At least 10 students in the class will sign up for a senior lire 

saving course at the conclusion of a unit on water safety. 

74. Seventy-five percent of the students will understand differential 

equations. 

75. Students will recite with no more than one error Milton s sonnet 
"On His Blindness". 

76. 60% of the students will prepare 500 work book reports on famous 
social scientists. 

77. The students will thoroughly comprehend at least 80/(, of the 
scientific theories treated in class. 

78. The students will paint a still-life study employing two point 
perspective and at least three colors. 

79. Everyone in class will orally recite a given Spanish dialog with 
no errors in pronunciation. 

80. Students will be able to match chemical compounds with their 
valences on a written test. 



PART V. CRITERION MEASURES 

Part V (a) Directions ; For the following list of five items distinguish 
between those which could be employed as educational criterion measures 
and those which could not be marking an A on your answer sheet for 
each item which could be used as a criterion to evaluate educational 
programs# Be careful to match the item numbers with the appropriate 
item on the answer sheet. 

81. Standardized achievement tests 

82. Locally constructed tests of pupils’ progress in spelling 

83. The age of learners 

84. An anon3Tmous self-report questionnaire which, among other more 
obvious purposssj contains a cjuGstion soliciting th6 respondent s 
values regarding labor unions 

85. The number of serious pencil marks, carvings, etc., on pupil 
desks which must be removed by maintenance personnel each 
summer 



o 
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Part V (_b) Directions : the four category scheme presented below, classify 

each of the following five lettered items by selecting the appropriate letter 
in the -pace for each item on the answer sheet. 



C lasses of Criterion Measures 

A. Learner-behavior--natural conditions 

B. Learner- behavior — manipulated conditions 

C. Behavior-product“-natural conditions 

D. Behavior-product--manipulated conditions 

86. Scores on the Kuder test of vocational interest 

87. Instances of pupil misbehavior during recess periods 

88. Surreptitious observations of learner behavior in nationally distributed 
"situational stress" tests involving accomplices 

89. "Courtesy" as reflected by adolescent boys giving their seats on the 
bus to women who might otherwise be obliged to stand 

90. Final extemporaneous speeches in senior English class 

PART VI. PROTOTYPE TEST ITEMS 

Part VI (a) Directions ; Mark an A on the answer sheet by the number of 
any objective which includes a statement of presentation conditions. 

91. The learner will compose in writing a four line verse. 

92. The student will compare Romanticism and Victorianism literary movements. 

93. The teacher will list five elements necessary in a particular geometry 
proof. 

94. Given pictures of four colored objects, the child will circle the object 
which is red. 

Part VI (b) Directions : For the following pair of objectives and items, 

mark A if the item corresponds to the objective in terms of response called 
for, directions , and presentation conditions . If the item does not correspond 
with the objective, mark B. For each objective and item you should make 
three responses on the answer sheet. 

Vf Vf ^ * 

Ob iective : The student will write the course of action most consistent 

with the tenets of good citizenship outlined in class when given a 
social problem not previously encountered. 



Item ; Choose a social problem you are familiar with and in less than 
300 written words describe how you would deal with it in terms of the 
citizenship concepts described in class. 

***** 



Student Response (95^ 



Directions (96) 



Presentation Conditions (97) 
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* * * * 

Ob iectivs l The student will be able to select all written notations 
which describe permissible moves in a chess game when presented with 
four choices. 

Item : For the opening move by white, which of 

are permissible? (Check your answers.) 

(a) K-KZ (b) P-QN3 (c) N-QB3 

* * * * 

Student Response (9S) Directions (99) Presentation Conditions J[100i 



PART VII. INSTRUCTIONAL SPECIFICATIONS 

Part VII (a) Directions : Indicate which component of the IS^ is most nearly 

described or identified by each of the following items by marking on your 

answer sheet: 

A. for "Terminal Behavior" 

B. for "Instructional Cue" 

C. for "Elicitor" 

D. for "Limits" 

E. for "Entering Behavior" 

101. Provides rules or procedures designed to improve learner performance. 

102. Contains prototype criterion items. 

103. Describes the stimulus conditions under which the criterion behavior 

will be demonstrated. u • 4 

104. Contains the information the learner requires to perform the criterion 

behavior. 

Part VTI (b) Directions: This portion of the examination consists of two 

terminal behavior is followed by potential components of the instructional 
specification. For each statement which is appropriate, to its terminal 
behavior, whether it is a complete component or not, mark an 'A' on your 
answer sheet. Nothing need be marked if the statement is not appropriate 

to the terminal behavior. 

No attempt has been made to be devious or to trick you with tests of your 
knowledge of subject matter pertaining to each objective. You should only 
concern yourself with the appropriateness of the component to the terminal 
behavior, not authenticity of the content. 

* * * * * 

"To construct an equilateral triangle, given the necessary equipment." 

105. Instructional Cue: "An equilateral triangle is a triangle in which 

all sides and angles are equal." 

106. Elicitor: "Name an object that has equal sides and equal angles. 

107. Limits (negative): "Triangles with less than 3 equal sides and 3 
equal angles." 



the following moves 

(d) QB-R5 

* 
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108. Entering Behavior: "To identify triangles, given examples of triangles, 

squares, and rectangles." 



"To write the longitude and latitude of any given position on a map, 
given a map with longitude and latitude lines. 

109. Instructional Cue: "Longitudes and latitudes are essential for 

locating positions on the open sea." 

110. Elicitor: "Degrees latitude on the position indicated are , 

degrees longitude are 



PART VIII. APPROPRIATE PRACTICE 

IX. KNOWLEDGE OF RESULTS 

X. PROMOTING LEARNER INTEREST 

XI. AVOIDING IRRELEVANCIES 

Parts VIII, IX, X, and XI. Directions: First, read the short self-instruction 

program dealing with statistics. (You may wish to respond to the program as 
though you were the intended learner.) Then answer the series of examination 

questions 



MADAM, MAY I PLEASE fiEASURE YOUR CENTRAL TENDENCIES? 

(A Non-Exemplary Program) 

by 

W . James Popham 



Program Objective : At the conclusion of the program the learner will 

be able to select the correct numerical values of th? mean, median, or mode 
from multiple choice alternatives when presented with a set of fictitious data. 
Enroute Objective : The learner will be able to match the terms mean, 

median, and mode with definitions of these measures. 

Enroute Obiective ; The learner will be able to compute the numerical value of 
the mean, median and mode from small sets of fictitious data. 

Prerequisite Behaviors ; As necessary entry behaviors, the learners 
should be able to read and perform the following operations with one, 
cwo, and three digit numbers: add, subtract, multiply, divide. 
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A. As an individual cor,..erned with the development of instructional 
materials, you will encounter situations in which you wish to describe 
how well a group o£ learners performed on a criterion test after com- 
pleting your instructional sequences. Ideally, this description should 
be supplied as parsimoniously as possible to save your time as 
those with whom you are attempting to communicate. One of the most ef 
ficient ways to describe a set of data is through the use of st atistica l 

measurss of central ten dency* 



B. Statistical measures of central tendency are numerical indicators 

of the manner in which the scores of individuals in a group of scores 

(such as test data) tend to cluster near the center of the seal- 
the scores are measured. The three measures to be treated in this 

program are the mean , the median , and the niode . 



C, The mean is calculated by adding together all of the scores in a 
set of scores (also called a distribution) and then dividing them by 
the number of scores in the set. For example, consider the following 
set of seven scores: (8, 7, 5, 4, 3, 1, 0). When these scores are 
added together, their sum is 28. Dividing 28 by 7, the number of scores 
we find that 4 is the mean. For the following set of four scores (10, 

8, 8, 2) then, we can see that 7 is: 

a. the mean c. the mode 

b. the median d. none of the above 

For these and subsequent items in which you are to make a choice or 
supply an answer, please make your response directly on the booklet, 
then read beyond the three asterisks. It may be necessary to mark off 
the section below the asterisks So that you do not inadvertently see 
the correct answer. For the question above, circle the letter of the 



o 



best answer. 
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Sipce 7 is derived by adding' all scores (28) and the;* ‘iLviding by the 
nupiber of scores (4), it is the mean. 



D, The median is the iioint which divides a set of scores into two equal 
halves. For instance, in the following set of scores (9, 9, 6, 5, 2) 
the number 6 splits the scores into two equal halves, hence it is the 
median. 



E, Sometimes the median is not an actual score. See if you can determine 
the median for the following set of scores,’ This is a difficult problem 
and if you get it right, you will be one of the eleven per cent who answer 
correctly. See if you can choose the right answer and avoid the 89% 
stigma! 

Scores; 7, 7, 6, 4, 2, 1 
Median choices (circle one): 

a. 6 

b. 5.5 

c. 5 

d. 4.5 

*;»• it 

If you chose answer C you are a member of the ’’lofty eleven,” To calculate 
the midpoint of this distribution, it would be necessary to interpolate 
between scores of 6 and 4 to obtain a 5, 



F, The mode is the most frequently appearing score in a distribution. Thus 
in the following set of scores (9, 8, 4, 2, 2, 1) the score 2 appears most 
frequently so it is the mode. What would the mode be for this set of scores 
(8, 8, 7, 7, 7, 2, D? 

a. 8 

b. 7 

c. 2 



